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Through consideration cf a conceptual framewcrk, this 
paper proposes that a comprehensive evaluation can be made that 
accounts fcr a project *s ability to accomplish social change. The 
concepts are; (t) ideological developEent--perinits the evaluator to 
predict the probability of the ultimate change a project can 
accomplish based on the change agents* consideration of social 
justice: (2) legacy of change — permits evaluators to determine the 
potential for continued change after a project's funding ends; (3) 
comparability — contrasts one project with another to determine the 
relative merit of each; and (tj) social efficacy—includes the 
determinaticn cf the effect of a project and the use of resources 
used by the prcject. This framework provides a better sta:adard of 
judgment than can be made from a single evaluation methodology, 
because the ccncepts are not discrete: consideration of each provides 
a triangulation of data and perspectives to evaluate outcomes of 
social intervention, A major benefit cf this framework is that it 
cffers a procedure for considering effects of social change projects 
that incorporates multiple perspectives in the analysis. Also 
considered, however, are which social elements must be changed to 
effect social change ultimately, (Author/GK) 
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A Conceptual Frameworic for the Evaluation of Planned Social Change 
KEN KEMPNER 

ABSTRAa 

Although methodologies have been developed in education to evaluate 
interventions that attempt to overcome injustices in education^ no single 
methodology cociprehensivel/ considers the multitude of variables nor the 
ultimate social effect of an intervention. Therefore, to evaluate an educa- 
tional change project^ a conceptual framework is developed to consider the 
ultimate effects of an intervention and the extent of these effects over time* 

The premise of this paper is that through consideration of the concepts 
of the frameworle (ideological developm^t^ change legac/^ con^arabilit/, 
social efficacy)^ an evaluation can be made that accounts for a project's 
ultimate ability to overcome social injustices* 



EVALUATIMG SOCIAL CHANGE 



When the justice of certain customs or institutions is not universally 
accepted by all members of society, dissatisfied individuals may develop 
strategies to overcome x^at they consider to be social injustices* To determine 
the overall effectiveness and worth to society of these strategies the outcomes 
of such attempts at social change should be evaluated* Because planned 
social change projects are complex social interventions and not controlled 
experimental treatments, the ability to determine a project's effect versus the 
effect of other social factors is extremely diffieult* To overcome this 
inherent difficulty in determining causality, a number of evaluation Tiiethodologies 
have been developed in the social sciences which focus on varying aspects of a 
change project's probable effects* 
A Multiple Perspectives Approach 

Although numerous evaluation methodologies exist, particularly in the 
field of educational change, no single evaluation methodology considers 
the multitude of causal variables nor the variety of target groups involved in 
interventions to overcome injustices in educational systems* In the evaluation 
of an educational change project, upon which this paper is based, a conceptual 
framework ^ developed that considered the project^s effects and the extent 
of these effects over time* 

Since there is no single perspective on how best to induce change, 
there can also be no one way to evaluate change* What the conceptual framework 
presented here offers, therefore, is a variety of perspectives to consider the 
ultimate effect of an intervention that attempts to overcome social injustices. 

Although most contemporary educational evaluation methodologies may con-- 
sider the social change project to be successful If It has raets its objectives. 
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the social effect of the project may still be unknown* The major premise of this 
paper is that through consideration of each of the concepts of the framevorle 
a comprdiensive evaluation can be made that does account for a project*s 
ability to accomplish social change. While the concepts are not discrete, each 
one considers a slightly different aspect of change that is needed to c^valuate 
the ability of a social intervention to alter individual or group behavior. The 
four concepts are: 1) ideological development, which allows the evaluator to 
predict the probability of the ultimate change a project can accomplish based on 
the change agents* consideration of social justice; 2) legacy of change, which 
allows evaluators to determine the potential for continued change after a 
project^s funding has ended; 3) comparability^ which contrasts one project or 
activity with another to determine the relative merit of each; and 4) social 
erficacy^ which includes the determination of the effect of a project and the 
use of resources employed by the project* From this conceptual framex^rle 
evaluators can have a better standard of judgment to determine if a change project 
has asi^isted the process of social change than can be made from any single 
evaluation methodology* 
Toward a More Comprehensive Methodology 

The th&oretical development in this paper is based upon the author^s 
work as one of two evaluators of a federally-- funded project on sex equity in 
public school administration* The data were collected over a two and one-half 
year period (between 1977 and 1979) and included the collection of data from within 
*.h2 project, data on the project's impact on the target groups, and the tabulation 
of delta from state departments of education in several western states* The project 
housed at a university in Oregon* 

Because the evaluation literature does not provide an effective 
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and comprehensive model for evaluating the ability of a change project to 
accomplish social improvement (see Kempner, 1979), the development of a 
methodology was necessary to evaluate the educational change project under 
consideration here. Since planned change projects are only one of many 
variables that can^ effect change in a complex social system, an effective 
evaluation must be complete, enough to capture the full reality of the change 
effort and attempt to determine the causal effect of a project. 

Because many evaluation methodologies do not consider the ultimate 
goals of a project nor the effect of a project over time> evaluation methods 
which provide a more complete description of reality are needed. .While present 
evaluation methodologies may consider a project to have been successful by 
meeting its objectives, the social effect of the project may still be unknown. 
The purpose of this paper is to present the framework for the evaluation of 
planned social change projects that was developed so that a larger amount of 
knowledge about the ultimate social effect of a project can be gained. To 
accomplish this task a conceptual framework, which is defined by Denzin 
(1970, p. 67) as a ^^road structure of both explicity and assumed propositions" 
composed of descriptive categories, was developed. 

Conceptual frameworks are useful because they can assist researchers 
and evaluators provide a more comprehensive analysis of social interventions 
than is typically allowed under more traditional forms of evaluation. Campbell 
(1969, p. 409) comments on this issue by stating, ''If the political and admin- 
istrative system has committed itself in advance to the correctness and efficacy 
of its reforms, it cannot tolerate learning of failure." The nature of program 
funding often prevents researchers and evaluators from questioning the 
ultimate project goals and questions, such as why the target group remains the 
target group, are rarely considered. Weiss (1975, p. 24) explains: 
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We concentrate attention on changing the attitude and 
behavior of target groups without concommitant attention to 
the institutional structure and social arrangements that 
tend to keep them "target groups." 

Through the development of conceptual frameworks, which attempt to identify 

the determinants of social change, evaluators will be able better to consider 

the full range of effects that are necessary for social improvement* 

THE COMCEPTUAL FRAMEWORK 

The following framework is provided, nof^as a theory from which 
uniformities can be derived (see Merton, 1968), but as an initial work from 
which further deductions may be made to arrive at a theory for the evaluation 
of social change. It is apparent from a critique of contemporary evaluation 
methodologies^ (Kempner, 1979) that a comprehensive theory of the evaluation of 
social change has yet to be developed. Although this paper does not attempt to 
develop such a theory, it does consider those social concepts that should be 
evaluated to determine if a planned change project is effecting social change* 
Ideological Development: The Concept 

Ideology can be explained as an "interlocking set of beliefs (Bowers, 1977, 
p. 35) used to justify a group's opinion or doctrines on socio-^political 
issues, such as cultural maintenance or change. Because individuals' ideology 
provides the'^guiding thread for their problems, their concept and their focus 
of thought" (Mannheim, 1936, p. 4), it is essential for evaluators to consider 
this first concept in the framework, the role of ideology in the development 
of strategies produced by change agents. The identification of a project's 
ideology can help explain why project members made certain decisions to pursue 
change in specific ways. The consideration of ideology is, therefore, the first 

I:R!c ^ 



5. 



of the four concepts in the framework for the evaluation of planned social 
change . 

Morality and Social Justice . In planned Change interventions the consideration 
of ideology by evaluators is important in determining what values guided the 
processes that were responsible for the outcomes that occurred. By emphasizing 
the investigation of cognitive processes, as Piaget does (1951), evaluators can 
attempt to understand the beliefs and ideology that influence the ultimate goals 
of a social intervention. Whereas Piaget has established a system for the 
description of intellectual development, Kohlberg has developed a theory of 
moral development which is quite useful in the evaluation of social change. In 
Kohlberg^s theory (1975, p. 675): "morality is a natural product of a universal 
human tendency toward empathy or role taking, toward putting oneself in the shoes 
of other conscious beings. It is also a product of a universal human concern for 
justice." 

Kohlberg*s theory provides an understanding of how moral reasoning 
changes through a developmental process. In this theory, individuals are seen 
by Kohlberg to fall into one of six stages of moral development, which extend 
from simple reciprocity at the first stage to a "conscience or principle 
orientation" at the sixth stage (Kohlberg, 1967> p. 171). Because higher moral 
reasoning is dependent upon the concern for justice, the dynamics of moral 
development can be applied to the development of a belief system or ideology of 
social justice. Therefore, in the conceptual framework presented here, moral 
development is considered comparable to ideological development. By using 
Kohlberg's basic concepts, evaluators can categorize a change project on its overall 
level of ideological development basad on the project members* consideration of 
social justice. 
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Because a limited aiiK)uiit of goods and services are available in society, 
the issue of social justice arises when the fairness of resource allocation is 
considered (see Uume, 1971, p. 71). Evaluators must, therefore, determine if 
change agents are actually seeking social improvement by changing the social 
aristocracies of resource distribution, or if the agents are oblivious to issues 
of social justice and are seeking only to better their own position. Because 
any behavior can be justified on the basis of beliefs or ideology, no universally 
accepted concept of justice has yet been devised. Even though a universal 
concept does not exist, effective evaluators must still judge a social change 
project^s members upon their definition of social justice to determine if social 
Improvement can be made under such a definition. Although justice is typically 
defined by the beliefs of the dominant culture of a society, Rawls (1971) does 
offer two basic principles which provide possible criteria for the evaluation 
of a ch^Tige project^s concept of social justice. 

According to Rawls 's theory, when individuals who are unaware or "veiled" 
to their own position In society and knowledge of their fortunes, abilities, and 
Intelligence assemble in an hypothetical "original position" to develop universal 
principles of justice, they would choose the principles of Rawls 's theory (1971, 
p. 12). This original position is defined by Rawls as the "appropriate initial 
status quo, and the fundamental agreements reached in it are fair" (p. 13). 
R^wls believes that the f.lrst principle to be chosen in this hypothetical original 
position would state, basically, that every Individual has an equal right to all 
basic liberties within the system of liberty for all. The second principle would 
explain that social and economic inequities are allowable only If such 
inequities benefit the "least advantaged" members of society (pp, 302-202), 

Although these principles can provide helpful criteria for the evaluation 
of planned change, these principles are still based upon certain cultural 
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assumptions made by Rawls (see Wolff, 1977). Even though Rawls^s theory is 
limited in its applicability and universality, the principles of justice can 
serve as guidelines in the determination of the extent of change a social project 
can or has induced. Members of a change proje^ need not subscribe to Rawls^s 
principles of justice to be effective, but the manner in which the concepts of 
the "least advantaged" and '*libefty"are considered can indicate how aware 
individuals are of their own biases. 

Stages of Development . Because morality is seen as the product of an 
individual's concept of justice in Kohlberg's theory, Pougiales and Kempner (1979) 
identify three stages of ideological development for planned social change projects. 
The levels are not intended to be discrete categories but provide a hierarchical 
range of ideology to distinguish between three general developmental stages. The 
assumption of ideological development is patterned after Kohlberg^s theory of 
moral development. Although Kohlberg^s wor^ has been with individuals, research 
has been Initiated on the moral atmosphere of groups (Power, 1978) to characterize 
a group^s level of moral development. On the basis of these investigations and 
their own work, Pougiales and Kempner have proposed the following levels of 
ideological development in planned change projects, which were used to 
evaluate the project under consideration. 

The first level of ideological development is analogous to Kohlberg's 
(1973, p. 164) "preconventional level," which includes the first two stages of 
development in his theory. At this level, change Is justified on an efficient, 
reciprocal basis, not a value one. Although a project goal may be desirable at 
this level, the rationale for its implementation is efficiency or self-interest. 
There is no real accomplishment of social change at this level, but rather a 
replacement of one social system for another — neither having much concern for 
social justice. 

er|c ^ 
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At the second level of development, change agents are characterized 
by an acceptance and reliance on the status quo. This level is patterned 
after Kohlberg's (1973, p. 164) "conventional level." Even though individuals 
at this level of ideological development are socially critical, the impetus for 
change comes from the dominant culture. The change effort is, therefore, compatible 
with present cultural norms and seeks to make adjustments within the social 
system through methods that can be condoned by the status quo. Justice at this 
level of development is defined by the beliefs and values of the dominant culture. 

Chan^ agents at the third level of ideological development are 

characterized by their cultural awareness beyond the simple acceptance of the 

inherent values of a society. Change is Justified by principles of social Justice 

which would overcome social aristocracies that perpetuate inequities. Change 

agents at this ideological level are morally similar to Kohlberg*s (1973, p^ 165) 

"postconventional , automomous, or principled level.'* Rather than seek only 

adjustments in the Inequities of the status quo, change agents at this level 

of development attempt to alter practices that perpetuate aristocracies. 

By seeking out alternative methods from among the variety of theories of change, 

individuals at the third level of development will utilize whatever methods will 

be most effective iii accomplishing social improvement. 

I 

Because effec^tive evaluation designs must assess the ultimate goals, the 
Justification for the goals, and the manner in which the goals were met, consideration 
of the ideological development of a change project and its members can provide 
an analysis of the process?,s that were responsible for a project's effect. 
By generating Information instead of verifying conclusions, evaluators can provide 
data to explain why an outcome occurred. Just as Piaget focuses upon cognitive 
processes to explain the reason for an outcome, the emphasis on the con- 
sideration of Ideological development is upon the processes of a change group to 
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determine why certain outcomes occurred. Through observation of how menibers 
of a change project develop and implement strategies^ evaluators can determine 
the level of ideological development and critical awareness of the group 
as a whole. 

Ideological Development: Evaluation of the Concept 

Because evauators cannot look at every aspect of a project, a sample of 
activities must be taken to "liinlt the elements studied" (Stake, 1967, p. 532). 
In a manner, similar to the sampling of transactions in the field of accounting 
(Mautz and Sharaf, 1961), events have been evaluated from the project under 
consideration that attimpt a representative picture of the project. 

By selecting elements or strategies that represent each of the four con- 
cepts in the framework, a comprehensive judgment can be made of a project^s 
ability to accomplish social improvement. Through the integration of both 
quantitative and qualitative methods, the conceptual framework can incorporate 
various perceptions of events in the evaluation. To explain how each of the 
concepts can be used iu the overall evaluation of a social change project » 
examples are presented after the explanation of each concept to show how the 
concepts were used and integrated in the evaluation of the project under 
consideration here. 

Rather than rely only upon questionaires, surveys, or interviews of 
intended behavior^ evaluators can assess more realistically a project^s level of 
development by observing the project member's behavior during the intervention. 
By focusing also on the processes which affect behaviors and outcomes^ evaluators 
canexplain how and why an event occurred^ instead of describing only what occurred. 
The data for this analysis were drawn from sources such as field notes, tapes of 
meetings, project documents and interviews. Specifically, the assessment of the 
project's ideology was made by the evaluators on the basis of the interactions 
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observed among the staff menibers and the target groups and on the basis of the 
patterns that developed in the use of certain str£itegies by the project. 

The project was organized around the pursuit of six gQals; 1) building 
awareness of sex inequity in education; 2) restructuring the content of training 
programs in education; 3) training and recruiting women for educational leadership; 

4) building capacity in the state for the continuation of equity activities; 

5) conducting research on sex inequities; and 6) disseminating the project^s 
outcomes. The basic focus of these goals was upon the development of methods 
to help women enter existing administrative positions. Only one activity was 
devoted to altering the present structure of administrative positions. By 
emphasizing integration and movement into jobs» instead of stressing ways to 
change the nature of these jobs» the project members accepted the basic structure 
of the administrative position. Because the project *s intervention strategies 
were aimed basically at getting more women into existing administrative positions^ 
strategies to alter the nature of the position were not emphasized. Strategies 

to move more women into administrative positions are much different than 
strategies aimed at changing the nature of the position that perpetuates male 
domination. The other project goals included strategies aimed predominantly at 
facilitating the movement of women into the existing administrative hierarchy* 

The outcomes of these studies^ although interesting^ are less important 
to the ultimate evaluation of the project than how the choice of these strategies 
indicated the project's pref ertedmethod of intervention. Aithougji these studies 
provided worthwhile data, the project's research efforts were limited primarily 
to awareness-building. Additionally, the decisions or preferences for which target 
groups were to be emphasized provided an indication of the project's acceptance 
of the structural status quo. 

Althougji the instrumental goals of the restructuring activities were met, 
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the eventual outcomes of the project^s efforts focused upon individuals already 
In power (professors and administrators) and upon dissemination of information, 
not restructuring administrative jobs. Even though project members did not accept 
the status quo of fewer women in educational administration, the strategies 
selected displayed a choice not to change the econotEiic and hierarchical 
nature of administrative positions in the educational system. While the 
restructuring activities did accomplish changes in the requirements for ascension 
to administrative positions, they did less to affect the nature of the position, 
or the behavior of the people already in thesej[>ositlons. 

Several actions by project members indicated their consideration of 
social justice, which provide an evaluation of ideological development. For 
example, the project did not generally consider what should be the ultimate 
goals of an intervention designed to erase sex inequities in administration. 
During a staff meeting on the project^s self**eva-uatlon, one project member 
objected to a suggested emphasis on social justice in the evaluation. This 
individual explained that the project was not funded to achieve that goal. A 
brief discussion ensued over the project^s ultimate objectives and although con- 
sensus was not reached, a number of project members agreed that the project^s overall 
goal was to increase the number of women in educational administration and 
not consider issues of social justice. 

The limitation of the project's ultimate objectives to equity in 
administration is analogous to the distributive justice of resources* Methods 
for the distribution of resources can be devised, but these methods do not 
assure justice nor assure the worth to society of the production of these goods* 
Similarly, methods have been devised to certify and employ school administrators 
and the project Intervened to provide equal access for women to the certification 
and employment process. To assume, however, that present administrative 
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positions are inherently worthwhile to society, regardless of the person 
who holds the position, is a limited viewpoint. Furthermore, the assumption 
that simply having more women in these positions will create greater social 
justice is also a narrow belief, especially if the nature of the position remains 
unchanged . 

A further limitation of the project in its ultimate goals was the 
limitied consideration of the least-advantaged members of the educational system**- 
the children. Because the ultimate purpose of schools is the education of 
child^*. 1, presumably the function of administrators and teachers is to help 
create the best possible environment in which children can learn* By choosing 
not to deal with the present educational structure and the role of administrators 
and teachers, the project did not consider how the social purpose of education 
could be met better. Although the attainment of sex equity is a worthy goal, this 
goal has meaning only if it benefits the least-advantaged meni>ers of the 
educational system in Rawls^s theory of justice. By seekjLng only to move women 
into the present structure of administration, the project focused its efforts on 
giving women access to a position of uncertain worth and social justice. 

Overall, the project was guided predominantly by what Coleman (1971, 
p. 643) terms a "psychologist's theory," This theory presumes that by changing 
the jjidividual, social change can be effected* Techniques employed in this 
individual-change process tend to be those which attempt changes in behavior by 
informing individuals of the social problems that exist. Through this method, 
change agents attempt to induce people to change their behavior. The primary 
focus of the project's strategies was upon the stimulation of individual 
awareness and an information on the dissemination of information as a change 
technique instead of more conf rontative strategics. For example, one proposed 
activity was to present course materials on sex equity in some of the training 
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sessions of the administrative extern program conducted by a department of 
the university. During the project's first year the staff menijers attempted 
to participate in the extfem program but were told the schedule was full. 
Rather than directly confront the administrators of the program* the staff 
mecfl^ers of the project pursued the issue no further and the project was 
never included in the extern program* s training sessions during the three 
years of the project. 

. Decisions not to pursue certain types of activities indicated the staff 
members' techniques of change; an integrative, internal approach that avoided 
confrontation. Because of this desire to avoid conflict and possibly 
alienate certain groups, the project chose strategies to increase these groups' 
awareness, rather than directly confront their behavior. Rossi and Williams 
(1972, p. ^21) are skeptical over the potential success of change programs, like 
this, "that are directed at transforming individuals who are relatively autonomous." 
They explain that "when individuals are the target of programs, it can be 
anticipated that success will be harder to achieve." Although it is easier, 
initially, to direct change efforts at individuals, unless there are also 
methods to induce changes In behavior, little ultimate change can be 
accomplished. 

Because of the techniques of change employed by the project and on the 
basis of the project members' general acceptance of the present structure of the 
administrative position, the ideology of the project can be classified at the 
second level of development. As discussed previously, individuals and groups at 
the second level of development are characterized by their reliance on the values 
of the dominant culture and their change efforts are typified by an acceptance 
of the predominant cultural norms and institutions. At this level of development 
change is considered only within the strictures of the present institutional 
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structures* Because the current social Ideology Is Implicitly acknowledged, 
the change accomplished is only what the system will allow* 

The project^s activities were predominantly information-oriented, 
^ile favoring collaboration and avoiding confrontation with institutional 
leaders* Although these activities were not accomplished without some 
effect, interventions that occur at this second level of development are 
limited in their potential for accomplishing significant social improvement* 
Change strategies, that are guided by a level-two ideology are much less 
extensive in their social effect than those strategies that are guided by 
a level-three ideology, which address methods of change beyond the inherent 
values of the society. 

The ideological classification of a change project, at a specific 
level, while providing one indication of a project's effect, indicates the 
underlying assumption of change that motivated the project's members 
during this project. Additional information is still needed by evaluators, 
however, to understand the social effect of a change program* Although the 
concept of ideological development can indicate the social improvement made 
by a project, evaluators must determine the potential for continued change of 
a project. 

Legacy of Change: The Conce:g^ 

Because planned social change projects are usually ephemeral in nature, 
the project's ultimate effects may not be apparent for serveral years after 
the funding has ended* Additionally, to create extensive social change, the 
ramifications of interventions should be apparent in the future through a continued 
legacy of the change effort. An example of this legacy is a strategy of the project 
under consideration that recruited women to doctoral programs in educational 
administration. Because the women who entered the program need several years 
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to complete their degrees and several more years to gain the necessary 
experience to tecome administrators, the effects of the project's recruitment 
strategy may not be apparent for six or seven years. Therefore, evaluation 
of the project only on the basis of its immediate outcomes would provide 
an incomplete portrayal of the ultimate change a project may induce. Because 
social change projects should leave a legacy after the intervention has ended, 
it is essential for evaluators to consider this second concept in the 
framework to estimate the future potential of continued social improvement 
a change effort will create. 

The need for a time perspective in evaluations is necessary because as 
Denzin (1970, p. 105) explains: '^Instruments that elicit only predispositions 
to act, and fail to follow the unfolding trajectory of action through time, 
run the risk of leaving analysis at a superficial level." To prevent such super-- 
ficiaX analysis evaluators must identify the processes that may assist or hinder 
a project from continuing its effect over time. Whereas the analysis of a 
project^s ideology provides an indication of the potential effect of ultimate 
change, continued social improvement is possible only if strategies which create 
a legacy of change are employed. 

Through the analysis of the procedures and the products change 
agents use or create to assure the continuation of their intervention 
efforts, evaluators can offer better estimations of the capability of a 
project to facilitate change in the future. Although social change projects 
may be successful in meeting short-term purposes the overall evaluation of 
a project *s success cannot be made tmtil the ultimate goals are considered 
over the long-term effect of the project. The determination of the legacy 
a project, will leave after it has ended can provide evaluators with an 
indication of the future potential for change a project can induce. Legacy 



18 



16. 



of change, in conjunction with the consideration of ideological development 
provides evaluators with an excellent understanding of a project^s ultimate 
ability to accomplish social improvement. 
Legacy of Change: Evaluation of the Concept 

Since the evaluation of a project's present outcomes cannot account for 
the future effects of a project, the potential of a project to leave a legacy 
of change must be considered. The project under consideration here developed 
a number of activities that will continue the process of change beyond the 
funding of the intervention. One of the major accomplishments of the project 
was the establishment of an organization which will continue the change effort 
initiated. An example of the project's legacy was the development of a state- 
wide organization of women administrators in education. Under the original 
sponsorship of the project, this group was organized to continue some of 
the major activities of the project. Specifically, the organization continued 
the publication of a newsletter, the coordination of a directory for adminis- 
trative candidates, and the organizational planning the project had provided 
(e*g., conferences, information clearinghouse)* Of th^ many project activities, 
the creation of this statewide organization did create the most significant 
legacy to continue the change effort beyond the life of the project. 

In the evaluation of a legacy, such as the organization created by 
the project, even though the change effort will continue, the ultimate effect 
of this activity must be considered. Although the organization is a major 
legacy of change created by the project, the extent of social change that can 
be accomplished by the newly developed organization may be limited by its level 
of ideological development. The success in creating the organization was a 
considerable achievement of the project, but the change approach of the organ- 
ization may limit its potential for altering the sexist nature of the admin^ 
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istrative structure. 

Because the organization is a product of the project^ its initial method 
of change operates within the values of the status quo. Even though the 
project operated at the second level of ideological development, the legacy 
of change it has lef the state-wide organizatiorv has the potential for 
inducing more change than had the project left no legacy at all. The project 
can be regarded as the first step in the developinent of an overall change 
strategy that has led to the organization of women administrators. For the 
organization to induce social change, however, it must utilize strategies 
that will dramatically alter the institutions that perpetuate the sex 
inequities in educational administration. 

In addition to the processes that leave a legacy of change, products 
may also^be created that endure beyond the life of a project. For the project 
under consideration the creation of products concerning issues of sex 
equity in education was a high priority. Numerous research articles, a 
research textbook, a slide-tape show, and several versions of a final report 
were produced by the project. These products will provide a continuing 
legacy to continue the intervention initiated by the project. Although 
the potential for these legacies to induce social change may be limited by the 
level of ideological development under which they were developed, the project^s 
legacies can continue to facilitate some change. 

Comparability: The Concept 

The concept of comparability, the third in the framework, is similar 
to "social evaluation theory," because as Pettigrew (l967, p. 243) explains, 
"human beings learn about themselves by comparing themselves to others . " 
Essentially, to understand value, an object or event must be considered in 
relation to something else; objects in isolation have no inherent worth with- 
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out referent value. As Winch and Campbell (1969, p. 143) observe: "Since 
it is central to the scientific endeavor to seek for relationships among 
variables, the paradigm of the scientific problem is that in which observations 
are classified with respect to one another." 

To determine the effect of a social intervention, evaluators must 
consider the impact of one strategy in relation to the impact of another 
strategy. Unless evaluators can offer advice to decision-makers based on a 
comparative analysis of strategies they have little basis for determining which 
strategy was the most beneficial for the decision-^makers and target groups. 
As Scriven (1974, p. 64) claims, "All useful evaluation is comparative...." 

While an experimental approach to evaluation has many benefits 
(Campbell and Stanley, 1963) such Intentional experiments are not always 
practical or ethical during social change interventions. AddltionaUy, 
directly comparable events or strategies may not always be available. Even 
though direct comparla<?i3s may not be feasible, by investigating the effects of 
related strategies or interventions evaluators can gain a better understanding 
of the real effect of a specific intervention. For example, although the 
various project strategies of the project under consideration could be 
contrasted with one another to offer a comparative analysis, there was no 
project identically comparable to the overall intervention. 
Comparability: Evaluation of the Concept 

As an example of how this concept was evaluated, a quantitative 
analysis is presented that was used to discover If the number of women admin- 
istrators in the state in which the project operated (Oregon) was significantly 
different than several other states during the same years. This analysis 
was conducted to indicate the overall effect of the project In Oregon compared 
to effects that might be occurring in other similar states. Although this 
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analysis cannot prove the effect of the project's Intervention, It cati 
determine If the changes In Oregon over several years are significantly 
different than the changes in other states. 

To test various models to determine the effect of year and of state, 
the "log-linear" analysis of contingency tables is used (Goodman, 1970, 1972). 
By gathering data from other states comparable to Oregon, a contingency table 
by state, year, administrative position, and sex tabulated (presented 
in the Appendi:i£) . Three western states (Colorado, California, Washington) 
were chosen as comparisons to Oregon. Although Colorado is most similar 
to Oregon in population, California and Washington are comparable in the 
regional interests these states share with Oregon. Becuase data on the 
basis of sex were available for a five year period, the analysis covers the 
0973-74 through 1977-78 school years. Administrative positions were divided 
into three categories: superintendents and assistants, principals and assistants, 
and a general category for directors, supervisors and coordinators. 

The log-linear analysis is useful because it predicts the "natural 
logarithm of the odds of falling into one category of the dependent variable" 
(^vers, 1976, p. 6). The COMTAB computer program (Nolan, 1974) was used 
to test the models summarized in Table 1. 

The first model tested predicts that each varialbe is independent in 
its effect. This model does not fit the observed data because the probability 
of being wrong in rejecting this hypothesis is zero (P" 0.0000). The second 
model, which predicts two-way interactions, also does not fit. The third 
model, three-way interactions, does fit the data. Because the data fit this 
tliird model reasonably well, other less complex interactions were tested to 
discover if a more single three-way interaction could fit the data. As the 
summary shows, no other model could fit the data as well as the third model 
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TABLE 1. — Summary of selected models 





■* Mortal - 


2 




p 


1. 


Year> Position^ State> Sex 


6820.691 


103 


0.0000 


2. 


Year-Position, Year-Sex, Year-State, 
Posit iort- Sex, Position-State, Sex- 


503.999 




0 0000 


3. 


Year-^Po sit ion-Sex, Year-Position-State, 




23 


) 

0 9473 


4. 


Year-Position-State, Year-Sex-State, 
Po si t ion— S State 


51 HQS 


31 


0 0130 


5. 


Year-Position-Sex, Year-Sex-State, 
Fosl t lon-Sex-St a t e 


DZ . D/O 


44 
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6. 


Year-Position-Sex, Year-Position- 
Dt.at.c, jrosiLtxon— DcX— Dtaue 


70 710 


33 


0 0001 


7. 


Year-Position-Sex, Year-Position- 






070000 


8. 


Year-Position-Sex, Year^Position- 


437 400 


39 


0 0000 
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9. 


Year-Position-Sex, Position-Sex- 


113.53? 


55 


0.0000 


10. 


Year-Position-Sex, Year-Sex-State 


415.967 


50 


0,0000 


11. 


Position-Sex-State, Year-Sex- State, 
I ear Jrosxt xon 




52 


0 0091 


12 


Year-Position-Sex, Year-State, 
Position-State, Sex-State 


471.044 


61 


0.0000 


13. 


Year-Sex-State, Year-Position, 
Position- 


444.624 


58 


0.0000 


14. 


Position-Sex-State^ Year-Position, 
Year-Sex, Year-State 


145.671 


63 


0.0000 


15. 


Year-Posi t ion-Sex-St ate 


0.000 


0 


-1.0000 



*The models are fitted to the contingency table in Appendix, 
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(the last model is expected to fit exactly, since it predicts all 
possible interactions) . Because the models of independent and two-way 
interaction did not fit the data, the analysis suggests that sex, year, 
state, or administrative position, alone, or in any pair are not sufficient 
to explain the effects in the whole model. 

The next step of the analysis is to investigate further the effects of 
the third model, which fits the data. Table 2 presents the natural logs 
of the cell effects and the standardized value of the effect (only the 
cells for women administrators that are relevant to the analysis are presented). 
The most important effect parameters in the analysis are those for the 
three-way interaction of year^sex-state. In the first year of analysis, the 
number of women administrators in California (cell effect = .131789) has 
the largest effect, followed by Colorado (.112469), Oregon (.095798), and 
Washington (-.340057). In the second year, Oregon has the largest effect, 
although it is not significant because the standardized effect equals only 
1.500, which is less than 1.96, the typical (.05) level of significance. 
This standardized effect is the ratio of the effsc^ divided by the standard 
deviation (Nolan, 1974, p. 80) and is, therefore, similar to the use of 
z^scores. 

Additionally, in this analysis the effect parameters also show no 
significant effects by state over the remaining years in the analysis. 
Since the effects in Oregon are of more concern, however, it is important to 
ncte that the effect of women administrators in the fifth year was in a large 
negative direction, which approaches significance ("1. 687268). In this year 
the number of women administrators in Oregon had a negative effect in the 
u^del; that is the relative frequency of women administrators in Oregon during 
the 1977-78 school year declined compared to the other states during that year* 
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TABLE 2. — Selected effect parameters of best-fitting model: Year-sex-state 









w X ^ a 


1973-1974 

California 


Up a1i ATy c^^nn 


Women 


Effect 
Standardize 


0.095798 
1.110114 


0.112469 
1,093859 


1974- 


0,131789 
2.000078 

■1975 


-0.340057 
-2.X33516 


Women 


Effect 
Standardize 


0.115620 
1.500487 


0.064084 
0.687689 




0.037765 
0.624399 


-0.217469 
-1.47422^ 










1975-1976 




Women 


Effect 
Standardize 


0.076900 
1.009219 


0.081480 
0.909152 


1976- 


0.019495 
0.327334 

■1977 


-0.17>876 
-1.217402 


Women 


Effect 
Standardize 


0.014703 
0.193930 


0.039432 
0.443988 


1977- 


0.050887 
^ 0.869027 

■1978 


-0.105022 
-0.730100 


Women 


Effect 
Standardize 


-0.303023 
-1.687268 


-0.297467 
-1.593665 




0.239935 
1.393052 


0.840424 
1.662150 



The best-fitting model Is all possible three-way Interactions* model number three In 
Table 1. ' 



* NOTE; Parameters for Men are Inverse of those for women. Since this Is the case In 
dlchotomous categories* the parameters for men are not Included In the table. 
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While this analysis does not mean that the project^s activities had a negative 
effect, it does suggest that the efforts of the project were not significant 
enough to overcome whatever social phenomena were occurring in the state to 
reduce the relative frequency of women administrators. 

In summary, the log-linear analysis shows that all possible three- 
' Way interactions among sex, state, y^^r, and adtoinistrative position are 
needed to explain the effects in the entire model. Specifically, the data 
suggest that the changes in the number of women administrators in Oregon 
compared to the women administrators in the other three states is not 
significant. Additionally, the data appear to show no . f j^nificant effects in 
Oregon during the years the project operated, 1976-1978 .(Data were not 
available from the other states for the final year of the project, 1978-79.) 
Although the intervention and some of its activities may have been necessary to 
facilitate the small positive effects shown, they have apparently not had an 
appreciably significant effect when compared to the concommitant effects in 
the other states surveyed. The findings would indicate that social factors 
other than the project, such as Affirmative Action or Title IX, may have 
been responsible for the differences. 

Social Efficacy: The Concept 

The final concept of the framework is, basically, a "so what?" 
question, which considers the worth to society of a project's efforts. The 
social efficacy of a project refers to its effect, extent of effect and the use 
of resources by the project. 

In the determination of a project's social efficacy, evaluators must ask 
if the intervention is worth doing, even if it is successful. The choice of 
a specific social intervetnion is always made at the expense of another 
method or use of resources. This "notion of costs as benefits forgone" 
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(Haller, 1974, 408) Is the opportunity cost sacrificed In favor of 
choosing a certain strategy over an alternative one. To determine the 
social efficacy of an intervention, evaluatorrs must consider, therefore, 
what was given up In favor of the chosen Intervention. Additionally, 
evaXuators must consider the use of resources by a project and the extent of 
effect caused by the Intervention. In this manner the determination of social 
efficacy can provide an evaluation of the worth to society of an intervention 
and:. the effectiveness of the resources employed. 

An Important component In the evalua^tion of social efficacy Is^ the 
efficient use of resources employed by change agents. Tlie social cost Involved 
must be compared to the benefits derived and evaluators must judge If the 
intervention was an efficient use of people, time, facilities, and money. 

Aa intervention must also be assessed according to its sufficient 
use of resources; that is, did the project adequately use all resources 
available? To acconq)llsh a social intervention effectively, change agents 
must identify huioan and material resources that can assist the change effort. 
Because interventions occur in a socl^environment, change ageftts will make 
contact with target groups who are both hostile and supportive to the change 
advocated. Regardless of the intervention methods used, change agents must 
work with individuals outside their group to facilitate the intervention 
process. Contacts must be made with the target group so change agents can 
understand the needs of these Individuals. This role-taking ability allows 
change agents to plan intervention strategies more in accordance with the needs of 
target groups. Sufficient contact must be made by change agents to develop an 
empathlc realatlonshlp with both hostile and supportive target groups. Change 
agents who Isolate themseleves from the target groups have little basis for 
knowing what change is really needed. 
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In addition to the sufficient use of human resources, a project should also 
be evaluated upon its sufficient use of physical resources. To accomplish 
ati intervention effectively, change agents must identify both human and 
material resources that are. available to assist the change effort. Just as 
people exist who can help change agents conduct an intervention, physical 
or material resources may also be available to assist this effort. To 
accomplish an efficient intervention, change agents must investigate what 
resources already exist that are pertinent to the change effort and then 
use these resources to prevent unnecessary duplication. From the findings 
of previous intervention efforts, change agents can plan innovative strategies 
that sufficiently consider both the knowledge gained and the resources 
developed or used during similar interventions. 

The final issue to be considered by evaluators in the detennination 
of the social efficacy of a project is the intervention's effect and the extent 
of this effect. Even though an intervention may constitute an efficient use of 
resources and have a good potential for future change, the intervention may cause 
little social improvement. To understand the social efficacy of an intervention, 
evaluators must determine if the intervention produced the intended effect. 

In addition to a project's effect, the extent of this effect, like its 
legacy of change, is an important consideration for evaluators. A project may 
develop processes and products to leave a legacy of change, but evaluators 
must still deteinaine if this legacy has the ability to produce an 
effect extensive enough to achieve social change. 
Social Efficacy; Evaluation of the Concept 

Tlie first example of how the project under consideration was evaluated 
with this concept concerns the project's sufficient u&e of resources. Although 
the project did coordinate its activities with several of the major insti- 
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tutional groups in the state, contact with teacher's organizations or with 
individuals in the administrative certification program who were not pursuing 
doctorates was not emphasized. Since women in education are found in the 
greatest numbers at the teaching level, these women compose a large group 
of potential administrators who have relevant experience. By emphasizing 
the recruitment of women to doctoral programs, the project did not 
sufficiently address all the women who could move into administration. 

Even though fewer physical resources "th^ human resources on sex 
equity were available to assist the project, the staff meitibers did more 
completely use existing resources. The project became a focal point for 
information and acted as a clearinghouse for resources on sex equity in 
the state and region. This important function was to be assumed by the 
organization of administrative women organized by the project. 

To be socially productive change agents must be efficient in their 
use of resources. Therefore, the second example of how the project was 
evaluated with the concept of social efficacy concerns the project's efficient 
use of resources. The major activity of the project was the establishment of 
a network to moaitor hirings within the state and to provide information 
on the availabiltiy of administrative jobs. Six coordinators were hired and a 
large data collection was initiated. Because of the expense of this network 
it appears, however, that a more efficient strategy could have been ixsed to 
yield similar results. As an example, instead of the expense of six 
coordinators traveling throughout the state, it is possible similar data could 
have been gathered over the telephone. Because the coordinators' data was 
information-oriented and not used as evidence to confront districts that did 
not follow equitable hiring practices, less complete data, collected at a 
lower cost, could have been a more efficient use of the project's resources. 
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Certainly the coordinators were able to compile comprehensive data by visiting 
school districts, but the high cost of data collection did not appear to justify 
the outcome* Although the data collection of the network was not an extremely 
efficient use of resources, the development of the organization of women admin-^ 
istrators was a much less expensive activity to implement and yielded more 
extensive results. 

Finally, to evaluate a change effort completely, change agents must determine 
if an intervention's effect is extensive enough to accomplish social change. Even 
though a project may successfully meet its goals, change agents have no assurances 
that the intervention will actually cause social change* In the project a number 
of activities were directed at changing the attitudes of both women and men on the 
inequities of educational administration. Of particular interest to the project 
was changing the attitudes of women toward themselves. This emphasis for the project 
was apparent in the recruitment of women to the doctoral program and in some of the 
workshop sessions conducted in the project-sponsored conferences. 

The need to encourage women to pursue administrative careers is, of course, 
paramount if women are ever to move into these positions. Strategies that only 
emphasize attitude change, however, may do little to actually change a woman's ability 
to become an administrator. A ''properly" dressed woman with a positive attitude of 
herself cannot hope to overcome an educational structure that discriminates against 
women, whatever her attitudes. Unless the institutional norms are changed to 
accept women, the only women that may become administrators are those who are most 
like the men already in power. 

The consideration of a project's effect and the extent of this effect provide 
an overview of an intervention's success. Even if a project has been sufficient 
and efficient in its use of resources, the effect of the project's activities may be 
minimal. Por example, the use of attitude strategies, as discussed above, illustrates 
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the slight effect of certain activities, even when these activities have been 
successfully accomplished. 

To detemtlne the extent of the project^s effect, the manner in which the 
intervention strategies changed the target group's beliefs and behaviors must 
be evaluated. The project^s emphasis on certain target groups was parallel to the 
project^s preference for particular strategies of change. Because one of the 
major goals of the project was to disseminante a model of change, the duplication 
of similar strategies was not the most efficient way to generate knowledge of the 
best methods for inducing change. 

The extent of the proJect*s effect was also limited by its adherence to 
a particular assumption of how change occurs. By not questioning their basic 
premises of how change can occur, the project's memb^ers failed to consider the 
potential effect of alternative assumptions of change. For example, when change 
agents operate primarily within the psychologist's theory of change, as this 
project did, only certain individual-oriented strategies are expected to effect 
change. By not using other methods of change based ou different theories, 
change agents do not test the potential outcomes that could be made from different 
techniques. When a project, whose goal is to generate knowledge of how to induce 
change, operates from the confines of one perspective, only limited change can be 
anticipated. When new solutions are not sought, the old institutional norms will 
not be significantly changed. 

Because the project did not employ multiple perspectives in the selection 
of change techniques, it did not fully investigate the efficacy of alternative 
methods of change. The project was ultimately limited in the extent of change it 
could produce because the majority of project members accepted the present structure 
of the administrative position; change is tolerated only when it is in accordance 
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with the values of the institution's members* 

While any single social intervention, such as this project, cannot typically 
accomplish improvement alone, a project's effect should be measured on its ability 
to add, incrementally, to the force of social change* Although the project helped 
organize women administrators in the state, the project was less effective in 
other areas of change* Organizational and awareness building strategies can be 
essential first steps in accomplishing social improvement, but further methods 
must be employed to induce non-supportive target groups to alter their behavior* 

OVERVIEW OF THE CONCEPTUAL FRAMEWORK 

By considering each of the four concepts of the framework evaluators can 
determine the ability of social chaxxge projects to accomplish change* Because the 
four concepts are not discrete, consideration of each provides a triangulation 
(Denzin, 1970) of data and perspectives to evaluate the outcomes of a social 
intervention* Since multiple perspectives exist in any social inquiry, from 
analysis of these four concepts pattertis of a project's effect can be discerned from 
different data and different perspectives « 

Consideration of each of the four concepts provides a more comprehensive 
evaluation than can be accomplished when only one method of evaluation is employed* 
A major benefit of this framework then Is that it offers a procedure to consider 
the effects of social change projects that incorporates multiple perspectives in 
the analysis* The framework also considers which social elements must be changed 
to effect social change ultimately* Since almost all change efforts are accomplished 
to overcome social inequities, evaluations should consider the ability of a change 
effort to alter the values, beliefs, and behaviors that perpetuate these inequities* 

Because this framework is not a theory "from which empirical uniformities 
can be derived" (Merton, 1968, p* 143) > It is proposed as a comprehensive scheme 
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that includes the major concepts that should be considered in the evaluation of 
change* Although further research is necessary to determine if the conceptual 
framework is inclusive and if it is effective in the evaluation of other social 
change projects, three of the most basic questions that must be considered will 
be addressed * 

One important question, which has arisen from the use of the framework, 
is the fairness in evaluating a project beyond its stated intents or abilities* 
A danger of this approach is the potential of the evaluator for establishing an 
unrealistic or Idealistic goal for a project* Such an artificial criterion for 
judgment can be avoided, however, by consideration of a project's level of 
ideological development in conjunction with the other three concepts in the 
framework* For example, should the menibers of a project adhere dogmatically 
to one method of change, whatever the project's goals, the amount of change will 
be limited by the assumptions of this method, since no one theory of change can predict 
uniform outcomes* 

Even though each change project confronts different social circumstances 
the evaluation should determine how efficiently and sufficiently a project employed 
its resources and considered the possible methods of change* To provide a fair 
evaluation, projects should be judged not upon intended goals or upon goals imposed 
by the evaluator, but upon the ultimate social goals of the intervention* 

A second question over the use of the framework is how the bias of the 
evaluator can be controlled* Because the evaluation of a project's level of ideology 
is the most subjective concept in the framework, the judgment of this concept must 
be triangulated with data from the other concepts in the framework if it is to have 
validity. Unless the evaluator's judgment can be supported by findings from the other 
concepts in the framework, the bias of the evaltiator will be exposed. 
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A third and basic question over the use of the framework is its applicability 



to a variety of planned social change efforts. Because all change efforts should 
consider the social justice of their activities^ the four concepts of the 
framework are appropriate equally for projects of different sizes and capabilities. 
Even though a small project caimot be expected to attempt a large variety of 
methods of change^ a project should be evaluated on its contribution to the change 
accomplished in the larger social movement with which it is associated* The 
excuse that a project was not funded to consider broader social issues does not 
relieve a project from the responsibility for consideraing the ramifications 
of its effects and the worth to society of such an intervention. 



(1871, p. 94-95) is to accomplish "social improvement" by facilitating the 
transition of a "custom or institution. . .from being a supposed primary necessity 
of social existence. . .into the rank of an universally stigmatized injustice and 
tyranny." Because an evaluation should consider the potential of an intervention 
to accomplish social improvement and determine if change has actually been induced, 
a theory is needed which can lead to uniform predictions of how change can be 
evaluated. Such a theory could assist in the development of strategies to overcome 
the inequities that exist in society because to evaluate change social scientists 
must know how change occurs. The framework presented here Is a step toward 
the development of such a theory. 
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••Contltigency table of public school admlnistraton by sex and by po^ltioa asslgi^Ttieiit la Ortz<m, Colorado, California, and Washington, 

1973-1978 



SuPerinteOdePte and Assistant-S^gerintendente Prindpala and Assistant PrinciPflls 



Females Males Females Males 





Or. 


Colo. 


Calif, 


Wash. 


Or. 


Cola 


Cftlit 


Wash, 


Or. 


Colo. 


Calif, 


Wash* 


Or. 


Colo, 


Calif, 


Wash, 


1973-1974 


4 


3 


31 


2 


311 


Z44 


1567 


365 


81 


105 


1426 


60 


1116 


1154 


7554 


1743 


1974- 197S 


6 


5 


^ 36 


2 


517 


250 


1612 


393 


82 


117 


1216 


86 


1164 


1196 


7970 


1773 


197S-1976 


4 


1 


42 


3 


324 


264 


1561 


370 


106 


157 


1283 


94 


1204 


1287 


7898 


1753 


1976-1977 


5 


1 


48 


4 


326 


265 


1578 


359 


120 


172 


1634 


127 


1204 


1300 


7555 


1752 


1977-1978 


3 


2 


44 




326 


261 


1595 




126 


189 


1830 


* 


1231 


1299 


7523 


* 



5itpervison, Directors, Coordinatois and Assistant 5iipervisocs, 
Assistant Directors, Assistant Coordinators 

Females Males 

Orgtoa Colorado CaHfomia Washiogton Oregon Colorado CalifCTnia Washington 



1973»1974 


160 


165 


551 


29 


474 


410 


2070 


396 


1974-1975 


214 


182 


657 


42 * 


470 


473 


^ 2238 


419 


1975-1976 


183 


209 


671 


54 


455 


515 


2203 


402 


1976-1977 


175 


222 


821 


71 


457 


507 


2292 


361 


1977*1978 


196 


192 


927 




477 


483 


2352 


* 



:4c 

Data not reported 

50URCE! Oregoa Certificated Peisotmel In Public 5chool Districts by sex and assignment, 1973-1^8, 5alem: Oregoa 5tate Department of 
Education; 5alaties and Related Information: Certificated Peiscimel, 1973-1978, Denven Colorado Department of Education; 5alaTics of 5uperan^ 
tendettts and Certain Other Adniinlstrative Officers in the Public 5choob of California, 1973-1978 , 5acranientO! California 5tate Department of 
Education; Certificated 5ta££ by Position and 5ex; 5tate 5ummaTies, 1972-1978, Olympia: Washington 5tate 5upcrintcndent of Public Instruction, 
Management Infomiation 5ervlce, 
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